SINGLE OR MULTI-PROJECTOR FOR ARBITRARY 
SURFACES WITHOUT CALIBRATION NOR RECONSTRUCTION 



FIELD OF THE INVENTION 



The present invention relates to a new approach for displaying an 
10 undistorted image on a surface of unknown geometry. 

BACKGROUND OF THE INVENTION 

15 

Recently, augmented reality has been undergoing a very significant 
growth. It is believed that three-dimensional (3D) video-conferencing, real-time 
annotation and simulation will be widely used in the near future [1]. Coupled with 
increasing computing power, the improvement of electronic frame grabbers and 

20 projectors adds even more possibilities. For instance, a virtual world could be 
displayed through projectors on the walls of a room to give someone a sense of 
immersion. Also, projection of an X-ray image over a patients body could help a 
physician to get more accurate information about the location of a tumour, or 
simple information about the patient's condition could be displayed in the 

25 physician's visual field. In many instances, many projectors have to be used to 
cover the whole environment or to prevent occlusions from people or objects. In 
addition, the projected images have to take into account the observer's position 
inside the room, and the projection surface geometry as illustrated in Figure 1. 
Still, immersing experiences are difficult to implement because image projection 

30 in various environments is hard to achieve. This is due to the wide range of 
screen geometry. 
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The projection problem can be divided into three main sub-problems. 
First, the image has to be corrected with respect to the screen geometry and the 
position of the observer. Second, multiple projector calibration and 
synchronization must be achieved to cover the field of vision of the observer. 
5 This includes colour correction, intensity blending and occlusion detection. Last 
of all, illumination effects from the environment have to be considered and 
corrected if possible. 

Many articles propose methods for solving parts of this problem in 
10 different contexts. Systems for projecting over non-flat surfaces already exist. It 
has been demonstrated that once the projectors are calibrated, texture can be 
painted over objects whose geometry is known [2]. This result has been 
confirmed with non-photorealistic 3D animations projected over objects [3]. 
Unfortunately, getting projector parameters is not always simple. For example, 
15 hemispherical lenses cannot be described with typical matrix formulation. Also, 
some applications need very fast surface reconstruction that remains today very 
challenging. Among reconstruction methods are structured light techniques, 
which can be used to scan small objects [4], [5] while stereo based systems with 
landmark projection over the surface offer a simple way to get the 3D geometry 
20 of the surface with triangulation. Of course, camera calibration is a prerequisite 
in each case [6]. 

When assumptions are made, simpler approaches can be used to correct 
the images. When the screen is assumed flat, keystoning rectification allows the 
25 projector and the observer to be placed at an angle relative to the surface [7, 8]. 
In this case, a camera and a tilt sensor mounted on the projector, or a device 
tracking the person is needed. Real-time correction is possible with video card 
hardware acceleration. 

30 Multi-projector systems are of two types. The first is a large array of 

projectors which require calibration and synchronization [9, 10]. As many as 24 
projectors can be used together to cover very large screens with high resolution. 
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In all cases, affine matrices are obtained for each projector during a calibration 
process with cameras. Intensity blending is later used to get uniform illumination 
over the surface. Effective methods to synchronize a large number of projectors 
are also presented by the authors. The second type of system does the same in 
5 the more general context of augmented reality where the surfaces are not 
necessarily flat. The process stays essentially the same except that surface 
reconstruction is needed. Intensity blending can then be used [6]. 

Real-time algorithms for correcting shades produced by a person or an 
10 object placed in front of a projector exist in the literature [11, 12]. The authors 
rely on other projectors to fix the image on the screen. 

Finally, some researchers were interested in the problem of colour 
calibration of multi-projectors [13]. They present a way of correcting the images 
1 5 according to the photometric characteristics of the projectors. 

SUMMARY OF THE INVENTION 

The present invention relates to a method of allowing at least one 
20 projector to display an undistorted, target image on a surface of unknown 
geometry, comprising: capturing, by means of a camera, an image of the surface 
from the point of view of an observer; establishing a mapping between pixels of 
the image from the camera and pixels of a projector image; projecting the target 
image on the surface using the projector, the projection of the target image 
25 comprising correcting the target image in relation to the established mapping to 
display on the surface a target image undistorted from the point of view of the 
observer. 

30 

The present invention further relates to a system for allowing at least one 
projector to display an undistorted, target image on a surface of unknown 
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geometry, comprising: a camera for capturing an image of the surface from the 
point of view of an observer; a producer of a mapping between pixels of the 
camera image and pixels of a projector image; the at least one projector for 
projecting the target image on the surface using the projector, the system 
5 comprising a corrector of the target image projected by the at least one projector 
in relation to the established mapping to display on the surface a target image 
undistorted from the point of view of the observer. 

10 The foregoing and other objects, advantages and features of the present 

invention will become more apparent upon reading of the following non 
restrictive description of an illustrative embodiment thereof, given by way of 
example only with reference to the accompanying drawings. 

15 

BRIEF DESCRIPTION OF THE DRAWINGS 
In the appended drawings: 

20 

Figure 1 is a top plan view of a setup including a screen, a camera and a 
projector and showing the respective positions of these screen, camera and 
projector with respect to each other; 

25 Figure 2 is a flow chart showing a series of operations conducted by the 

illustrative embodiment of the method according to the present invention, during 
an image construction process corrected for a specific projector-observer-screen 
configuration; 

30 Figure 3a is an example of projection patterns for bits b = 4, 3, 2, 1, used 

to obtain a function R b s \ 
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Figure 3b is an example of projection patterns for bits b = 4, 3, 2, 1 , used 
to obtain a function R b f \ 

Figure 4a is an example of histogram of A values representative of pixel- 
5 by-pixel differences between an image and its inverse, showing that large stripes 
yield very good separation; 

Figure 4b is an example of histogram of A values representative of pixel- 
by-pixel differences between an image and its inverse, showing that small 
10 stripes are hard to differentiate; 

Figure 5 is a graph of the percentage of usable pixels recovered from 
different stripe widths, wherein the bit number represents the lowest bit used in 
the encoding and the number of usable pixels decreases as the low order bits 
15 get used until merely none can be found, and wherein the maximum percentage 
value approximates the camera image coverage by the projector; 

Figure 6 is schematic diagram showing the projection of the center of 
S p (s 0t t 0 ) approximated by averaging the pixel positions that were not rejected, 
20 i.e. from S c {s 0 , t 0 ), wherein the mapping of the center of S p {s 0t t 0 ) onto the 
approximation of the center of S c (so, to) is a sample of R' 1 \ 

Figure 7 is a method of finding the value R" 1 (s* t*) of an undefined point in 
the projector domain, by interpolating the values from the enclosing triangle 
25 using barycentric coordinates; 

Figure 8 illustrates an image reconstruction process wherein, when the 
projector displays the corrected image, the camera image contains a copy of the 
source image; 

30 
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Figure 9 is a top plan view of a multi-projector setup including a screen, a 
camera and two projectors and showing the respective positions of these 
screen, camera and projectors; 

5 Figure 10a is a side view of two planes of a screen angularly spaced 

apart by approximately 60°, for a single projector setup; 

Figure 10b shows that undistorted image for the projector results of 
distorted image in the camera image, for a single projector setup; 

10 

Figure 10c is a corrected image wherein the curved line of brilliant pixels 
is a result of a different surface material, for a single projector setup; 

Figure 10d is a polar coordinates checkboard corrected pattern, for a 
1 5 single projector setup; 

Figure 10e is an enlarged view showing errors in the corrected image for 
a single projector setup, wherein black squares are area resulting of holes in the 
mapping and distortion are due to interpolation at borders with great 
20 discontinuity; 

Figure 11a is a side view of a screen showing a region covered by the 
projectors, for a multi-projector setup; 

25 Figure 11b is a checkboard without correction displayed by the first 

projector, for a multi-projector setup; 

Figure 11c is a checkboard without correction displayed by the second 
projector, for a multi-projector setup; 

30 

Figure 11d is a corrected image projected by the first projector, for a 
multi-projector setup; 
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Figure 11e is a corrected image projected by the second projector for a 
multi-projector setup, wherein errors in the right part of the image appear on a 
very inclined region of the dodecahedron with respect to the projector and 
5 wherein there is also a very large gap between the dodecahedron and the other 
screen surface; 

Figure 11f is a corrected image resulting from the combination of the 
images displayed by the two projectors of the multi-projector setup; and 

10 

Figure 11g is another checkboard pattern corrected image resulting from 
the combination of the images displayed by the two projectors of the multi- 
projector setup. 

15 

DETAILED DESCRIPTION OF THE ILLUSTRATIVE EMBODIMENT 

The following non-restrictive description introduces an illustrative 
20 embodiment of the method and system according to the present invention 
allowing one or more projectors to display an undistorted image on a surface of 
unknown geometry. To achieve this, according to the illustrative embodiment: 

- a signal camera is used to capture the viewer's perspective of the projection 
surface; 

25 - no explicit camera and projector calibration is required since only their 
relative geometries are computed using structured light patterns; 

- there is no specific constraint on the position or the orientation of the 
projector(s) and the camera with respect to the projection surface, except 
that the area visible to the camera must be covered by the projector(s); 

30 - the calibration is represented as a function establishing the correspondence 
of each pixel of a projector image to a pixel of the camera image; and 
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- after the mapping of each projector has been carried out, one can display an 
image corrected for the point of view of an observer, which takes into 
account the observer's position, the surface position, the projector position 
and orientation. 

5 

These method and system automatically take into account any distortion 
in the projector lenses. Typical applications of these method and system include 
projection in small rooms, shadow elimination and wide screen projection using 
multiple projectors. Intensity blending can be combined with this method and 
10 system to ensure minimal visual artefacts. The implementation has shown 
convincing results for many configurations. 

More specifically, the illustrative embodiment of the method and system 
according to the present invention (hereinafter the illustrative embodiment) 
introduces an image correction scheme for projecting undistorted images from 
the point of view of the observer on any given surface. The illustrative 
embodiment of these method and system exploits structured light to generate a 
mapping between a projector and a camera. The following description then 
shows how the method and system can be used for a multi-projector system. 

SINGLE PROJECTOR 

In order to project an image on a screen, some assumptions are made. In 
general, the screen is considered flat and the projector axis perpendicular to the 
25 flat screen. Thus, minimal distortions appear to an audience in front of the 
screen. Notice that the assumptions involve the relative position and orientation 
of the observer, the screen and the projector (see Figure 1). Those constraints 
cannot be met when an arbitrary projection surface (screen) of unknown 
geometry such as 10 in Figure 1 is used. In this case, information about the 
30 system including the screen 10, a camera 11, a projector 12, and the viewer 
(observer not shown in Figure 1) has to be determined dynamically to correct the 
projected images to avoid distortion. The approach commonly used starts by 
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20 
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finding a first function between the observer (not shown) and the screen 10 and 
a second function between the projector 11 and the screen 10. This involves 
calibration of the camera 1 1 and projector 12 described as: 
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where = implies equivalence up to a scale factor. The projective points (s, t, 1) T , 
(u, v, 1 ) T and (x w , 1) r are the projector image coordinates, camera image 

coordinates and surrounding world coordinates, respectively. 3D world points 

10 are related to projector image points by a 3x4 matrix F p and are related to 
camera image points by a 3x4 matrix F c . In the present method, the camera 1 1 
models the observer's view point. The world coordinates are known from 
landmarks located on a calibration object. Then, the image coordinates are 
identified by getting the position of those landmarks in the camera image and the 

15 projector image. After that, F p and F c can be used to reconstruct the screen 
geometry using a combination of structured light and triangulation [6]. 

Another approach, limited to a flat projection surface, involves the use of 
homographies to model the transformations from image and projector planes to 
20 the screen. The main advantage of homographies is their representation by 3x3 
invertible matrices defined as: 
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where (x Sl y s , 1) r is an image point in the screen coordinates system, (s, f, 1) r is 
an image point in the projector coordinates system, and (a, v, 1) r is an image 
point in the camera coordinates system. From H p and H Ct a relation between the 
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coordinates system of the projector and the coordinates system of the camera 
can be established as: 
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This mapping is invertible [14]. Homographies provide a linear mapping and are 
not directly useful when the screen is non-flat. Instead, the illustrative 
embodiment proposes to bypass the relation H P H C ^ by a piecewise linear 
mapping function R relating the camera and the projector directly. If R is 
10 invertible, it is possible to compensate for an arbitrary observer-camera-projector 
setup. 

The operations of the whole method conducted by the illustrative 
embodiment can be illustrated by the flow chart of Figure 2, comprising a pattern 
15 projection 21, an image acquisition 22, a bit identification 23, a mapping 
construction 24, an inverse mapping 25, and an image reconstruction 26. 

Operation 21: Pattern projection 

20 Structured light is commonly used in the field of 3D surface 

reconstruction. Using calibrated devices and a mapping between the camera 11 
and the projector 12, reconstruction can be achieved quite easily [4]. However, 
the illustrative embodiment will show that even though the camera 1 1 and the 
projector 12 are not calibrated, a mapping between the camera 11 and the 

25 projector 12 is feasible as long as no full 3D reconstruction is needed. 

For example, simple alternate black and white stripes can be used to 
build a correspondence from a point of the camera 11 to a coordinate in the 
projector 12 one bit at a time. For instance, for a n-bit coordinate encoding, each 
30 bit b (b e {1, ... , n}) is processed individually and yields an image of coded 
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stripes, each of width 2°~ pixels. The concatenation of all bits provides the 
complete coordinates. 

Figure 3 gives an example of the coded projector images. Many coding 
5 schemes are possible. Some try to increase noise resistance (grey code), and 
some other try to reduce the number of patterns in order to speedup the 
scanning process (colors, sinus). In the illustrative embodiment of the method 
according to the present invention, the simplest possible pattern is used. As 
illustrated in Figure 3, this simple pattern consists of two sets of horizontal and 
10 vertical stripes encoding s and t coordinates. 

If a partial knowledge of the projector and camera relative position is 
available, then a single stripe orientation can be derived from the epipolar 
geometry. Assuming no such knowledge, two orientations to accommodate 
1 5 arbitrary geometries are used. 

Operation 22: Image acquisition 

In order to compute a mapping function R from (u, v) to (s, t) this mapping 
20 function is first decomposed into partial mapping functions R b s and R b \ mapping 
the bit b of the s and t coordinates, respectively. These mapping functions are 
built by observing with the camera the projection of the corresponding stripe 
image and its inverse. Stripe identification is done with pixel-by-pixel difference 
between the image and its inverse, yielding A s and A* values between -255 and 
25 255. 

Figure 4 gives examples of histograms of A values. From these 
histograms, we find a rejection threshold r that is going to be used to define 
which values are usable or rejected. Although several approaches could be used 
30 to select automatically the threshold r, an empirical value of 50 is used. This is 
possible because the method is designed to tolerate rejected points. From this 
threshold, values are classified into three groups: 0, 1, and rejected (see 
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Equation 1). This test preserves only the pixels for which we can tell with 
confidence that the intensity fluctuates significantly between the inverse images. 

Point rejection occurs for two main reasons. First, a camera point might 
5 not be visible from the projector. Second, the contrast between the inverse 
stripes is too small. This occurs when the screen color is too dark or because of 
the limited resolution of the camera. In that case it causes two stripes of different 
colors to be projected onto the same camera pixel. This especially happens at 
borders between stripes. Thus, as the number of alternate stripes gets higher, 
10 the number of rejected pixels increases so much that we have observed that bit 
1 and 2 (1 and 2 pixels stripes) are generally useless (see Figure 4b). 



Operation 23: Bit identification 
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The bit mapping R b s can now be defined as: 



R(u,vy b = 



0 if A s (u 9 v)<-r 
1 if A s (w,v)>r 
rejected otherwise 



0) 



where b is the bit number and A s values are the inverse vertical stripes 
20 difference. Exactly the same process using horizontal stripes defines R(u, v) b 
from a' values. When a pixel is rejected, it will not be used anymore for the rest 
of the algorithm. 



Operation 24: Mapping construction 

25 



To obtain a complete mapping R from camera 11 to projector 12, the bit 
function R b s and R b are concatenated to get: 
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if R(u,v)i e {0,1} V6e {l,...,*} a*rf *(*,v)£ e {0,l} V 6 e{l,..,*}. (2) 

rejected otherwise 



As mentioned before, acquiring partial functions R(u, v) b s and f?(u, for 
low order bits b is generally impossible and Equation 2 rejects points for almost 
5 all (u, v) coordinates. Starting from the highest order bit, we observe in Figure 5 
that the percentage of usable pixels drops as lower order bits are used. 
Unfortunately, at some point, the percentage drops significantly (in Figure 5: 
below bit three). Consequently, the illustrative embodiment uses a mapping on a 
number of bits n' sufficiently small so that the number of usable points is not too 
10 small compared to the highest percentage. The n - n' unused bits are set to 0 
yielding a new mapping R' defined as: 

i?'={/?^..^_ w . +1 0...0,^...^_ n , +1 0...0} (6) 

1 5 The following section explains how to rebuild the function R' 1 from R'. 

Operation 25: Inverse mapping 

For all pairs of coordinates (s 0 , t 0 ) with bits from 1 to n-n' set to zero, 
20 there is defined a set of camera pixels S c (s 0 ,to) = {(u, v) \ R'(u, v) = (s 0 Jo)}- This 
is a contained region of the camera image as the thresholding eliminates 
possible outliers. We also define S p (s 0 ,t 0 ) = {(u,v) \ (u,v) = (u 0 ,v 0 ) mod 2 n ' n ), 
which is a 2 n ' n ' x 2 n ' n ' square of pixels in the projector image. It is generally hard 
to establish the exact correspondence between points of S c (s 0 ,to) and S p (s 0 ,to)- 
25 However one can estimate the projection center of S p (s 0 ,to) by taking the 
average of the points of S c (s 0f t 0 ) (see Figure 6). Now, one can make the 
assumption that the latter is mapped through R onto the center S p (s 0 ,to)- 
Applying this process for all non-empty S c for all (s 0 ,to) defines an under- 
sampling of the function R, and thus of K 1 as well. 
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In order to complete the construction of R' 1 t an interpolation scheme is 
used. The regular structure of the sampling makes it easy to implement because 
samples around a given point are easily found. If one of the samples needed for 
5 interpolation is undefined, interpolation also yields an undefined value. Whatever 
the value of n' is, the reconstruction of R" 1 takes the same amount of time. For a 
1024 x 768 image rebuilt with seven bits, this grid of points has dimensions 128 
x 96, representing 12065 squares for the approximation of the function. 
Selecting the right interpolation scheme can be tricky. For instance, a straight 
10 bilinear interpolation has no simple geometrical interpretation. To achieve 
piecewise planar approximation of the surface, each rectangle were divided into 
four identical triangles. To find an undefined value R* 1 (s*, t*) in the projector 
domain, the values from the vertices of the enclosing triangle are interpolated 
using barycentric coordinates (see Figure 7). 

15 

Operation 26: Image reconstruction 

Once the inverse mapping function R' 1 is found, the construction of the 
projector image is easy. We need to build an image in the camera 11 which 

20 corresponds to what the observer should see: the target image. This is done in 
four steps: i) Identification of the portion of the camera image that is covered by 
the projector; ii) Cropping of that portion in order to get a rectangular image with 
the same ratio as the source image; iii) Scaling of the source image into this 
rectangle; all other pixels are set to black; iv) Determination of the color of each 

25 point (s, t) of the projector by looking at R~ 1 (s, t) in the target image. The process 
is summarized in Figure 8. 

MULTI-PROJECTORS 

30 
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Addition of more projectors to cover a larger screen 70 (Figure 9) is 
rather simple. The method described above supports an arbitrary number of 
simultaneous projectors (Projector 1 and Projector 2 of Figure 9). A scheme for 
intensity blending must however be developed for an arbitrary number of 
5 projectors. Ideally, every point of the projection surface visible by the camera 71 
should be reached by at least one projector (Projector 1 and Projector 2 of 
Figure 9). Clearly, in this case, less points of the camera image are used for 
each projector and the algorithm must be adjusted accordingly. In particular, the 
number of bits recovered for R could be smaller. To expect good results, a 
10 higher resolution camera 71 is required when each projector only covers a small 
part of the camera image. As an alternative, the number of used bits n' can 
adjusted accordingly. 

One function fT 1 is recovered for each projector, one at a time. To 
15 provide a corrected image without holes, the projector images must overlap, 
resulting in unwanted intensity fluctuations. These could be effectively corrected 
by intensity blending algorithms. Whatever the number of projectors is, it should 
be made sure that the camera 71 sees the entire screen 70. 

20 

EXPRIMENTAL SETUP 



Even if the implementation does not depend on the projector or camera 
25 resolution, the quality of the results increases with the resolution of each device. 
In the experiments, a Sony Digital Handycam DCR-VX2000 (720 x 480 pixel 
resolution) and a Kodak DC-290 (1792 x 1200 pixel resolution) were used. In 
most cases, acquisition time is proportional to the resolution of the camera. 
Calibration time of each projector using the video camera was below two 
30 minutes and about 20 minutes for the Kodak digital camera. Two DLP projectors 
were used for the multi-projector setup: a Projectiondesign F1 SXGA and a 
Compaq iPAQ MP4800 XGA. Like every system using structured light, the 
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optical characteristics of each device itself limit the possible screen shape that 
can be reconstructed. For instance, the depth of field of both camera and 
projector restricts the geometry and size of the screen. After the calibration 
process is carried out, the image correction can be done in less than a second, 
5 but can be easily done in real-time on current video hardware technology. 

RESULTS 

Single projector setup: 

10 

Figure 10 illustrates how the image of one projector is corrected for a two- 
plane surface consisting of two circular screens. The camera and the projector 
were placed together so that the angles to each screen were about 50° and 70°. 
On the discontinuity between the two surfaces in the projector image, the 
15 distance along projection rays from one plane to the other was up to 15 
centimetres (Figure 10a). The Kodak camera was used so the R function could 
be constructed on eight bits out of 10. This allowed high precision corrected 
images (Figures 10c and 10d). Although errors are still present (Figure 10e), this 
resulted in very high precision corrected images (Figures 10c and 10d). 

20 

Multiple projector setup: 

The second test demonstrates how a multi-projector setup can correct 
occlusions on a very peculiar surface geometry. Here, projection was done on a 

25 dodecahedron in front of a flat screen (Figure 11a). Occlusions occur from both 
projectors, but very little from both simultaneously (Figures 11b-c). The Sony 
video camera was used and seven bits could be identified to compute R" 1 for 
both projectors. Results are shown in Figures 1 1d-g. Notice that even though the 
distortions and occlusions were large, the corrected images (Figures 11f,g) 

30 feature very few artefacts. 



CONCLUSION 
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The illustrative embodiment allows arbitrary observer-projector-screen 
geometries. Relying on a robust structured light approach, the method according 
to the illustrative embodiment is simple and accurate and can readily be adapted 
5 to multi-projector configurations that can automatically eliminate shadows. 

Algorithmic determination of the rejection threshold r and of stripe width 
could automate the whole process. It would also make it possible to have these 
parameters adapt across different regions of the screen resulting in better 
10 reconstruction. Acquisition time could be decreased using improved patterns. 
Furthermore, hardware acceleration of video cards could be used to boost the 
speed of the construction of function R" 1 as well as the corrected image 
generation. This would allow real-time applications where slides or movies are 
projected over moving surfaces. 

15 

Although the present invention has been described in the foregoing 
specification by means of illustrative embodiments, these illustrative 
embodiments can be modified at will within the scope, spirit and nature of the 
subject invention. For example, the projector(s) could be replaced by video 
20 screens, for example LCD or plasma screens forming the surface on which the 
image has to be formed. 
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